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TITLE OF THE INVENTION 
HIERARCHICAL PACKET SCHEDULING METHOD AND APPARATUS 

CROSS REFERENCE TO RELATED APPLICATIONS 
This application claims priority under 35 U.S.C. § 
119(e) to provisional patent application serial number 
60/042,943 filed April 4, 1997. 

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR 

DEVELOPMENT 
Not Appl i cabl e 

BACKGROUND OF THE INVENTION 

In any single scheduler methodology, such as the Packet 
Fair Queuing (PFQ) class of schedulers (described below) 
which include Virtual Clock (VC) , Self -Clocked Fair Queuing 
(SCFQ) , Start-time Fair Queuing (STFQ) , Minimum Start Time 
Clocked Fair Queuing Plus (MSTCFQ+) , Deficit Round Robin Fair 
Queuing (DRRFQ) , Packetized Generalized Processor Sharing 
(PGPS) , Worst Case Fair Weighted Queuing (WF 2 Q) and Worst - 
case Fair Weighted Fair Queuing Plus (WF 2 Q+) the scheduler is 
unable to fairly and efficiently handle a plurality of 
diverse traffic types or sessions. 

VC schedulers monitor the average transmission rate of 
statistical data flows and provides every flow with a 
guaranteed throughput. Each user requests an allocation for 
the amount of traffic the user expects. The packets are then 
time stamped and placed in a queue according to the value of 
their time stamp. VC schedulers have a tight delay bound 
that is independent of the number of sessions involved in 
order to provide for delay minimization. However, VC 
schedulers do not allocate available bandwidth as fairly as 
other schedulers . 
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Self Clocked Fair Queuing (SCFQ) schedulers as well as 
Start Time Fair Queuing (STFQ) schedulers have a high degree 
of fairness for allocating bandwidth. However, they do not 
have the desired delay minimization since the virtual time 
is no.t strictly monotonically increasing, their delay bounds 
are proportional to the number of sessions involved. 

The MSTCFQ+ scheduler determines the virtual finish time 
of the packet by determining the transfer time required for 
the packet from its length and' transfer rate, and adds this 
transfer time to the virtual- start time of the packet. The 
packet with the smallest virtual finish time is then 
scheduled for transfer. The MSTCFQ+ scheduler provides an 
effective tradeoff between performance and complexity as 
compared to other schedulers. 

DRRFQ schedulers have a high degree of fairness but are 
less effective when load balancing or distributed queues are 
used. 

The WF 2 Q and WF 2 Q+ schedulers have both a high degree of 
fairness and also the desired delay minimization. Therefore 
the WF 2 Q and WF 2 Q+ schedulers are not only fair but also have 
a tight delay bound that is independent of the number of 
sessions involved. However, the implementation costs of WF 2 Q 
and WF 2 Q+ schedulers are high compared to other schedulers 
since the WF 2 Q and WF 2 Q+ schedulers must perform two searches 
on each session; one search to determine the" packet start 
times of the first packet in each session, and a second 
search to determine the packet start times of all backlogged 
packets in each session; whereas other schedulers typically 
perform a single search on the first packet of each session 
for determining the packet start times. 

PGPS schedulers are similar to the WF 2 Q and WF 2 Q + 
schedulers, especially when utilizing packets having a small 
packet size, however the PGPS scheduler is better at 
minimizing worst-case session delay. 

The diverse sessions provided to the scheduler may 
comprise voice, video or data transmissions, each having 
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different Quality of Service (QoS) requirements and 
characteristics . There exists six classes regarding the QoS 
of traffic on networks such as an Asynchronous Transfer Mode 
(ATM) network. These six classes are Constant Bit Rate 
(CBR) , real time Variable Bit Rate (rtVBR) , non-real time 
Variable Bit Rate (nrtVBR) , Available Bit Rate (ABR) , 
Unspecified Bit Rate Plus (UBR+) and Unspecified Bit Rate 
(UBR) . Each QoS class has different requirements which must 
be taken into account by the scheduler in order to provide 
the maximum efficiency for handling this class of traffic. 
As an example, the CBR class is typically used for high 
quality video, the rtVBR class for voice and video, the 
nrtVBR class for low-quality video, the ABR class for data 
with flow control, the UBR+ class for data with no flow 
control and the UBR class for low priority data such as E- 
mail . 

Networks such as ATM networks are becoming more 
sophisticated and are expected to handle multiple connections 
organized as distinct sessions with various QoS requirements. 
It would be desirable to have a scheduler which can be 
implemented in a cost effective manner and handle the QoS 
requirements of different sessions fairly and efficiently. 



A hierarchical scheduler includes a first level of 
schedulers that matches a plurality of sessions having 
various Quality of Service (QoS) requirements with different 
schedulers which are best suited for a particular QoS class. 
A second level scheduler then schedules the outputs from 
certain ones of the first level schedulers. A third level 
scheduler schedules the remaining outputs the first level 
schedulers with the outputs from the second level scheduler 
to provide a hierarchical scheduler output. The hierarchical 
scheduler apportions bandwidth among sessions having 
different requirements in a fair and efficient manner 
according to the QoS requirements of the respective sessions. 



BRIEF SUMMARY OF THE INVENTION 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
The invention will be more fully understood from the 
following detailed description taken in conjunction with the 
accompanying drawings in which: 
5 Fig. 1 is a block diagram of the hierarchical scheduler 

of the present invention; and 

Fig. 2 is a flow chart illustrative of the method 
employed in the hierarchical scheduler of the present 
invention. 

10 

DETAILED DESCRIPTION OF THE INVENTION 
An hierarchical scheduler 10 in accordance with the 
present invention is shown in Fig. 1. The hierarchical 
scheduler 10 comprises an input 15, a first level of 

15 schedulers 17, a second level scheduler 80 and a third level 

scheduler 90 for providing an output 100. The input 15 
comprises packets from a plurality of sessions 12, 14, 16, 
18, 23 and 24 which have different operating characteristics. 
The input sessions are grouped into classes according to 

20 their Quality of Service (QoS) which will be described in 

greater detail below. A first level of schedulers 17 
comprises different types of schedulers 20, 30, 40, 50, 60, 
and 70. The outputs of a group of the first level schedulers 
32, 42, 52, 62, and 72 are provided to a second level 

25 scheduler 80. The output of second level scheduler 80 is 

provided to a third level scheduler 90. The third level 
scheduler 90 also receives the output 22 of one of the first 
level schedulers 20. While one output from a first level 
scheduler is shown connected to the third level scheduler, 

30 it: should be appreciated that any number of first level 

scheduler outputs could be connected to the third level 
scheduler, as well as any number of first scheduler outputs 
connected to the second level scheduler. The third level 
scheduler 90 provides an output 100. 

35 T k e presently disclosed hierarchical scheduler is 

particularly useful in an ATM network since the hierarchical 
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scheduler adapts the user traffic to the cell based network. 
The hierarchical scheduler supports different types of 
traffic at its input such as voice, video and data. The 
transmission requirements of voice, data and video are quite 
different, and each transmission type needs to be handled in 
a different manner in order to ensure the maximum efficiency 
and performance within such networks . 

Voice and low quality video transmissions can more 
readily handle errors. If a single voice or low quality 
video cell is distorted the quality of the voice or video 
transmission is not severely affected, as long as the 
incidence of cell distortion is low and the flow is 
substantially continuous. In fact, a voice or video cell can 
occasionally be discarded without severely affecting the 
quality of the transmission. For example, cell loss of a 
duration up to approximately sixteen milliseconds is not very 
noticeable or disturbing to the listener or viewer. Voice 
and video transmissions must be substantially continuous 
despite the occasional distorted or discarded cell, and 
therefore require a short queue length in order to reduce 
delay. The short queue length can occasionally overflow with 
a resulting loss of packets. On the other hand, voice or 
video transmissions cannot tolerate excessive network delay 
since extensive delay may result in the voice or video 
transmission becoming incomprehensible to the listener or 
viewer. 

In contrast to voice and video transmissions, data 
transmissions have a low or zero tolerance for errors since 
one distorted bit can change substantially change the data. 
Conversely, data transmissions have a high tolerance for 
network delay. Accordingly, data transmissions require the 
queue lengths to be longer to prevent packet loss in an 
overflow condition. 

More specifically, the hierarchical scheduler need to 
provide session management in order to properly and 
efficiently manage the bandwidth of an ATM network. Each 
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session must be individually serviced by the hierarchical 
scheduler in a fair and equitable manner. Session servicing 
by the hierarchical scheduler must provide appropriate delays 
and acceptable traffic losses according to the 
5 characteristics and requirements of each of the sessions. 

The hierarchical scheduler assures that delay-sensitive 
sessions such as video sessions are serviced within 2 
milliseconds, that voice sessions are serviced so as not 
allow the loss of traffic to exceed 1 to 10 percent of the 

10 samples, and that data sessions should not experience any 

data loss. Additionally, not only do , the characteristics 
among types of transmissions -such as voice, video and data 
vary, but the characteristics within a transmission type 
itself may also vary. Accordingly, these differences also 

15 need to be taken into account by the hierarchical scheduler. 

For example, a conventional 64 kbits per second voice call 
operates at approximately 166 cells per second, while a 32 
kbits per second call operates at approximately 3 8 cells per 
second. For maximum fairness and efficiency these 

-° differences must also be considered. 

Referring again to Fig. l, there exist six QoS traffic 
classes on ATM networks . These six classes are Constant Bit 
Rate (CBR) , real time Variable Bit Rate (rtVBR) , non-real 
time Variable Bit Rate (nrtVBR) , Available Bit Rate (ABR) , 

25 Unspecified Bit Rate Plus (UBR+) and Unspecified Bit Rate 

(UBR) . Each QoS has different requirements which must be 
considered by a scheduler. 

The CBR class 12 has very stringent cell loss and delay 
requirements and requires the scheduler to provide a quality 

'° of service equivalent to that of a circuit in a circuit 

switching network. Accordingly, the CBR class connects 
directly to a CBR shaper 2 0 and the output 22 of the CBR 
shaper 20 is connected to the highest priority input of the 
priority element 90. The CBR shaper 20 is essentially a 

:5 large First In First Out (FIFO) buffer which protects the 

network from bursty traffic - The output of the CBR class may 
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include short bursts of traffic which are buffered by the CBR 
shaper and released at a controlled rate so as not to exceed 
downstream bandwidth limitations. 

The sessions belonging . to the rtVBR class 14 have 
5 stringent delay requirements but can tolerate certain levels 

of cell loss. Certain guarantees of delay bounds and non- 
zero cell loss ratios are associated with this class. 

Sessions of the nrtVBR class 16 have less stringent cell 
loss requirements than the rtVBR class, but can tolerate 
10 certain levels of delay. This QoS also provides for 

guaranteed non-zero cell loss objectives. 

Sessions of the ABR class 18 can tolerate certain levels 
of delay and loss. Associated with this QoS are guaranteed 
minimum throughput or minimum cell rates. Additionally, ABR 
15 sources . may also adjust their transmission rates from time 

to time as required by a standard flow control algorithm 
implemented at each network node along the path of the 
connection . 

Sessions of the UBR+ class 23 are similar to those of 

2 0 the ABR class 18 but they do not involve network flow 

control. The UBR class 24 of .sessions do not have any 
specific loss, delay, or throughput requirements. 

As an example, the CBR class is typically used for high 
quality video, the rtVBR class for voice and video, the 
25 .nrtVBR class for low-quality video, the ABR class for data 

with flow control, the UBR+ class for data with no flow 
control and the UBR class for low priority data such as E- 
maii . 

As shown in Fig. 1, for rtVBR sessions 14, a Minimum 

3 0 Start Time Clocked Fair queuing Plus (MSTCFQ+) scheduler 3 0 

is used. The MSTCFQ+ scheduler determines the virtual finish 
time of a packet within a session by determining the transfer 
time required for the packet from its length and transfer 
rate, and adds this transfer time to the virtual start time 
35 of the packet. The virtual start time of the packet is set 

to the system virtual time when the packet was received. The 
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packet with the smallest virtual finish time is then 
scheduled for transfer. The MSTCFQ+ scheduler, due to its 
effective tradeoff between performance and complexity as 
compared to other schedulers, provides the stringent .cell 
delay requirements required by the rtVBR session. 

A Weighted Round Robin (WRR) scheduler 4 0 is used for 
scheduling the nrtVBR packets since this class does not have 
stringent delay requirements and only expects the network to 
guarantee their specified minimum throughput, while being 
inexpensive to implement. For the same reasons, a second WRR 
scheduler 50 is used- to schedule ABR packets and a third WRR 
scheduler 60 is used to schedule UBR+ packets. 

For the UBR class 24" a simple round robin (RR) scheduler 
70 is used since all the UBR sessions should be treated 
equally. Further, this scheduling hierarchy gives the 
network operator the flexibility to allocate any amount of 
bandwidth to support the UBR connections. RR schedulers are 
also inexpensive to implement. 

At the second level of hierarchy a second MSTCFQ+ 
scheduler 80 is used. This scheduler 8 0 schedules between 
the first MSTCFQ+ scheduler output 32, the WRR schedulers 
outputs 42, 52 and 62, and the RR scheduler output 72. The 
MSTCFQ+ scheduler is again utilized due to its effective 
tradeoff between performance and complexity as well as being 
inexpensive to implement as compared to schedulers offering 
similar performance. 

The static priority scheduler 90 gives the output 22 
from the CBR shaper 2 0 the highest priority and the rtVBR, 
nrtVBR, ABR, UBR+- and UBR as a group priority below the CBR 
class. The CBR shaper 2 0 is responsible for per-Virtual 
Circuit (VC) shaping and is work non- conserving in that the 
CBR shaper 2 0 will allocate bandwidth whether or not it has 
cells pending. As such, the CBR shaper 2 0 wastes bandwidth 
if no cells are pending. Accordingly, the CBR class is given 
the highest priority while the delay sensitive queues are 
given the next highest priority. 
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Referring to Fig. 2 the method of a particular 
embodiment of the present invention is shown. In the first 
step 110 packets providing a plurality of sessions are 
received. The sessions comprise various types of 

5 transmissions such as data, voice and video and may include 

flow-control . 

The next step, 120 results in the plurality of sessions 
being grouped into classes according to the QoS requirements 
of the respective sessions. Each session is grouped into one 
10 of the CBR, rtVBR, nrtVBR, ABR, UBR and UBR+ classes. 

The next step 130 provides shaping of the CBR class. As 
discussed above, the .CBR shaper releases traffic at a 
controlled rate so as to exceed downstream bandwidth 
limitations. 

15 As illustrated in step 140, a first MSTCFQ+ scheduler 

is utilized to schedule an output from the rtVBR sessions. 
As depicted in steps 150, 160 and 170 the nrtVBR sessions, 
the ABR sessions and the UBR+ sessions are scheduled with 
respective WRR schedulers. As shown in step 180 an RR 

20 scheduler is employed for scheduling an output from UBR 

sessions. 

As illustrated in step 190 a second MSTCFQ+ scheduler 
is employed to schedule an output from among the first 
MSTCFQ+ scheduler, the WRR schedulers and the RR scheduler. 
25 The next step 2 00 provides the CBR shaper output and the 

output of the second level scheduler to a static priority 
scheduler. 

Finally, in step 210 the CBR shaper output and the 
second level scheduler output are prioritized to provide the 
30 hierarchical scheduler output. 

The hierarchical scheduler of the present invention thus 
provides for management of various sessions having various 
requirements and characteristics in a fair, efficient and 
cost effective manner such that multiple sessions can be 
35 scheduled in a cell based network such as an ATM network. 
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Having described a preferred embodiment of the invention 
it should be apparent to those of ordinary skill in the art 
that other embodiments and variations of the presently 
disclosed embodiment incorporating these concepts may be 
5 implemented without departing from the inventive concepts 

he rein disclosed. Accordingly, the invention should not be 
viewed as limited to the described embodiments but rather 
should be limited solely by the scope and spirit of the 
appended claims. 
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CLAIMS 



We claim: 

1. A method for scheduling packet transmission in a network 
communication device comprising the steps of: 

receiving a plurality of packets, wherein each of said 
packets is associated with one of a plurality of 
classification types ; 

segregating said received packets into a plurality of 
sessions, each of said sessions having at least one of said 



forwarding said packets for each of said plurality of 
sessions to inputs of a corresponding plurality of first 
level schedulers, each of said schedulers having said input 
for receiving packets within respective sessions and an 
output, and wherein each one of said schedulers employs one 
of a plurality of first level scheduling protocols; 

receiving packets at inputs of a second level scheduler 
from the outputs of at least some of said plurality of first 
level schedulers, wherein said second level scheduler further 
includes an output and is operative in accordance with a 
second level scheduling protocol; 

forwarding packets from the output of said second level 
scheduler to a first input a third level scheduler, and 
forwarding packets from the output of at least one of said 
first level schedulers to a second input of said third level 
scheduler, wherein said third level scheduler has an output 
and is operative in accordance with a third level scheduling 
protocol ; and 

forwarding packets from the output of said third level 
scheduler in accordance with said third level scheduling 
protocol . 

2 . The method of claim 1 wherein said segregating step 
comprises the step of segregating said packets into a 
plurality of sessions wherein each of said sessions is 
associated with one of said classification types, and each 



10 



plurality of classification types; 
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of said sessions includes packets associated with at least 
one of a plurality of Quality of Service (Q 0 S) levels. 

3 . The method of claim 2 wherein said communication device 
5 comprises an Asynchronous Transfer Mode (ATM) switch and said 

plurality of Q Q S levels comprise Constant Bit Rate (CBR) , 
real time Variable Bit Rate (rtVBR) , non-real time Variable 
Bit Rate (nrtVBR) , Available Bit Rate (ABR) , Unspecified Bit 
Rate + (UBR+) and Unspecified Bit Rate (UBR) Quality of 
10 Service levels. 

4. The method of claim 1 wherein each of said first level 
schedulers, said second level scheduler and said third level 
scheduler comprise one of a CBR shaper, a Minimum Start Time 
15 Clocked Fair Queuing Plus (MSTCFQ+) scheduler, a Weighted 

Round Robin (WRR) scheduler, a Round Robin (RR) scheduler and 
a Static Priority (SP) scheduler. 



20 



5. The method of claim 1 wherein said classification types 
comprise CBR, rtVBR, nrtVBR, ABR, UBR+, and UBR Q D S levels. 



6. The method of claim "5 wherein said first level 
schedulers comprise a CBR shaper scheduler, an MSTCFQ+ 
scheduler, a plurality of WRR schedulers and at least one RR 

25 scheduler. 

7 . The method of claim 6 wherein said second level 
scheduler comprises an MSTCFQ+ scheduler. 

30 8. The method of claim 7 wherein said third level scheduler 

comprises a Static Priority scheduler. 

9 . The method of claim 8 wherein said CBR shaper receives 
at the input of said shaper packets associated with said CBR 
35 classification type. 
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10. The method of claim 6 wherein said first level MSTCFQ+ 
scheduler receives at the input of such scheduler, packets 
associated with said rtVBR classification type. 

5 11. The method of claim 6 wherein said plurality of WRR 

schedulers comprise first, second and third WRR schedulers. 

12. The method of claim 11 wherein said inputs of first, 
second and third WRR schedulers receive at respective inputs 
10 -packets corresponding to said nrtVBR, ABR and UBR 

classification types . 

13 . The method of claim 1 wherein said second level 
scheduling protocol comprises one of said first level 
15 scheduling protocols. 

14. The method of claim 10^ wherein said second level 
scheduling protocol comprises the same scheduling protocol 
associated with the rtVBR- classification type. 

2 0 

15. The method of claim 1 wherein said plurality of packets 
comprise voice transmissions, video transmissions or data 
transmissions . 

2 5 16. A method for scheduling packet transmission in a network 

communication device comprising the steps of: 

receiving a plurality of packets, wherein each of said 
packets is associated with one of a plurality of Quality of 
Service (QOS) levels, said Q Q S levels comprising Constant Bit 

30 Rate (CBR) , real time Variable Bit Rate (rtVBR) , non-real 

time Variable Bit Rate (nrtVBR) , Available Bit Rate (ABR) , 
Unspecified Bit Rate + (UBR+) and Unspecified Bit Rate (UBR) 
Quality of Service levels; 

segregating said received packets into a plurality of 

35 sessions, each of said sessions having at least one of said 

plurality of Q Q S levels, - 
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forwarding said packets for each of said plurality of 
sessions to inputs of a corresponding plurality of first 
level schedulers, each of said schedulers having said input 
for receiving packets within respective sessions and an 
output, and wherein each one of said schedulers employs one 
of a plurality of first level scheduling protocols, said 
first level scheduling protocols comprising a CBR shaper, a 
Minimum Start Time Clocked Fair Queuing Plus (MSTCFQ+) 
scheduler, a Weighted Round Robin (WRR) scheduler, a Round 
Robin (RR) scheduler and a Static Priority (SP) scheduler; 

receiving packets at inputs of a second level scheduler 
from the outputs of at least some of said plurality of first 
level schedulers, wherein said second level scheduler further 
includes an output and is operative in accordance with a 
second level scheduling protocol comprising a MSTCFQ+ 
scheduler; 

forwarding packets from the output of said second level 
scheduler to a first input a third level scheduler, and 
forwarding packets from the output of at least one of said 
first level schedulers to a second input of said third level 
scheduler, wherein said third level scheduler has an output 
and is operative in accordance with a third level scheduling 
protocol comprising a Static Priority Scheduler; and 

forwarding packets from the output of said third level 
scheduler in accordance with said third level scheduling 
protocol . 
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Receive packets associated with a plurality of sessions 
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